{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "medieval-neighbor",
   "metadata": {},
   "outputs": [],
   "source": [
    "BRANCH = 'v1.0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "developmental-gibraltar",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
    "\n",
    "Instructions for setting up Colab are as follows:\n",
    "1. Open a new Python 3 notebook.\n",
    "2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
    "3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
    "4. Run this cell to set up dependencies.\n",
    "\"\"\"\n",
    "# If you're using Google Colab and not running locally, run this cell\n",
    "\n",
    "# install NeMo\n",
    "!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "second-lucas",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you're not using Colab, you might need to upgrade jupyter notebook to avoid the following error:\n",
    "# 'ImportError: IProgress not found. Please update jupyter and ipywidgets.'\n",
    "\n",
    "! pip install ipywidgets\n",
    "! jupyter nbextension enable --py widgetsnbextension\n",
    "\n",
    "# Please restart the kernel after running this cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "challenging-pioneer",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo.collections import nlp as nemo_nlp\n",
    "from nemo.utils.exp_manager import exp_manager\n",
    "\n",
    "import os\n",
    "import wget \n",
    "import torch\n",
    "import pytorch_lightning as pl\n",
    "from omegaconf import OmegaConf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "employed-ethiopia",
   "metadata": {},
   "source": [
    "In this tutorial, we are going to describe how to finetune BioMegatron - a [BERT](https://arxiv.org/abs/1810.04805)-like [Megatron-LM](https://arxiv.org/pdf/1909.08053.pdf) model pre-trained on large biomedical text corpus ([PubMed](https://pubmed.ncbi.nlm.nih.gov/) abstracts and full-text commercial use collection) - on the [NCBI Disease Dataset](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3951655/) for Named Entity Recognition.\n",
    "\n",
    "The model size of Megatron-LM can be larger than BERT, up to multi-billion parameters, compared to 345 million parameters of BERT-large.\n",
    "There are some alternatives of BioMegatron, most notably [BioBERT](https://arxiv.org/abs/1901.08746). Compared to BioBERT BioMegatron is larger by model size and pre-trained on larger text corpus.\n",
    "\n",
    "A more general tutorial of using BERT-based models, including Megatron-LM, for downstream natural language processing tasks can be found [here](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb).\n",
    "\n",
    "# Task Description\n",
    "**Named entity recognition (NER)**, also referred to as entity chunking, identification or extraction, is the task of detecting and classifying key information (entities) in text.\n",
    "\n",
    "For instance, **given sentences from medical abstracts, what diseases are mentioned?**<br>\n",
    "In this case, our data input is sentences from the abstracts, and our labels are the precise locations of the named disease entities.  Take a look at the information provided for the dataset.\n",
    "\n",
    "For more details and general examples on Named Entity Recognition, please refer to the [Token Classification and Named Entity Recognition tutorial notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb).\n",
    "\n",
    "# Dataset\n",
    "\n",
    "The [NCBI-disease corpus](https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/) is a set of 793 PubMed abstracts, annotated by 14 annotators. The annotations take the form of HTML-style tags inserted into the abstract text using the clearly defined rules.  The annotations identify named diseases, and can be used to fine-tune a language model to identify disease mentions in future abstracts, *whether those diseases were part of the original training set or not*.\n",
    "\n",
    "Here's an example of what an annotated abstract from the corpus looks like:\n",
    "\n",
    "```html\n",
    "10021369\tIdentification of APC2, a homologue of the <category=\"Modifier\">adenomatous polyposis coli tumour</category> suppressor .\tThe <category=\"Modifier\">adenomatous polyposis coli ( APC ) tumour</category>-suppressor protein controls the Wnt signalling pathway by forming a complex with glycogen synthase kinase 3beta ( GSK-3beta ) , axin / conductin and betacatenin . Complex formation induces the rapid degradation of betacatenin . In <category=\"Modifier\">colon carcinoma</category> cells , loss of APC leads to the accumulation of betacatenin in the nucleus , where it binds to and activates the Tcf-4 transcription factor ( reviewed in [ 1 ] [ 2 ] ) . Here , we report the identification and genomic structure of APC homologues . Mammalian APC2 , which closely resembles APC in overall domain structure , was functionally analyzed and shown to contain two SAMP domains , both of which are required for binding to conductin . Like APC , APC2 regulates the formation of active betacatenin-Tcf complexes , as demonstrated using transient transcriptional activation assays in APC - / - <category=\"Modifier\">colon carcinoma</category> cells . Human APC2 maps to chromosome 19p13 . 3 . APC and APC2 may therefore have comparable functions in development and <category=\"SpecificDisease\">cancer</category> .\n",
    "```\n",
    "\n",
    "In this example, we see the following tags within the abstract:\n",
    "```html\n",
    "<category=\"Modifier\">adenomatous polyposis coli tumour</category>\n",
    "<category=\"Modifier\">adenomatous polyposis coli ( APC ) tumour</category>\n",
    "<category=\"Modifier\">colon carcinoma</category>\n",
    "<category=\"Modifier\">colon carcinoma</category>\n",
    "<category=\"SpecificDisease\">cancer</category>\n",
    "```\n",
    "\n",
    "For our purposes, we will consider any identified category (such as \"Modifier\", \"Specific Disease\", and a few others) to generally be a \"disease\".\n",
    "\n",
    "Let's download the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "federal-beads",
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR = \"DATA_DIR\"\n",
    "os.makedirs(DATA_DIR, exist_ok=True)\n",
    "os.makedirs(os.path.join(DATA_DIR, 'NER'), exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "relevant-juvenile",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Downloading NCBI data...')\n",
    "wget.download('https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/NCBI_corpus.zip', DATA_DIR)\n",
    "! unzip -o {DATA_DIR}/NCBI_corpus.zip -d {DATA_DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "radical-castle",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want to see more examples, you can explore the text of the corpus using the file browser to the left, or open files directly, for example typing a command like the following in a code-cell:\n",
    "\n",
    "! head -1 $DATA_DIR/NCBI_corpus_testing.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "specified-maine",
   "metadata": {},
   "source": [
    "We have two datasets derived from this corpus:  a text classification dataset and a named entity recognition (NER) dataset.  The text classification dataset labels the abstracts among three broad disease groupings.  We'll use this simple split to demonstrate the NLP text classification task.   The NER dataset labels individual words as diseases.  This dataset will be used for the NLP NER task.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affected-numbers",
   "metadata": {},
   "source": [
    "## Pre-process dataset\n",
    "A pre-processed NCBI-disease dataset for NER can be found [here](https://github.com/spyysalo/ncbi-disease/tree/master/conll) or [here](https://github.com/dmis-lab/biobert#datasets).<br>\n",
    "We download the files under {DATA_DIR/NER} directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "present-interference",
   "metadata": {},
   "outputs": [],
   "source": [
    "NER_DATA_DIR = f'{DATA_DIR}/NER'\n",
    "wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/train.tsv', NER_DATA_DIR)\n",
    "wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/devel.tsv', NER_DATA_DIR)\n",
    "wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/test.tsv', NER_DATA_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "identical-figure",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -lh $NER_DATA_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "powerful-donor",
   "metadata": {},
   "source": [
    "Convert these to a format that is compatible with [NeMo Token Classification module](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification_train.py), using the [conversion script](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/data/import_from_iob_format.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "utility-wesley",
   "metadata": {},
   "outputs": [],
   "source": [
    "! mv $NER_DATA_DIR/devel.tsv $NER_DATA_DIR/dev.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "suited-jenny",
   "metadata": {},
   "outputs": [],
   "source": [
    "wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/data/import_from_iob_format.py')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sensitive-victoria",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python import_from_iob_format.py --data_file=$NER_DATA_DIR/train.tsv\n",
    "! python import_from_iob_format.py --data_file=$NER_DATA_DIR/dev.tsv\n",
    "! python import_from_iob_format.py --data_file=$NER_DATA_DIR/test.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "graphic-debate",
   "metadata": {},
   "source": [
    "The NER task requires two files: the text sentences, and the labels.  Run the next two cells to see a sample of the two files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sound-surgeon",
   "metadata": {},
   "outputs": [],
   "source": [
    "!head $NER_DATA_DIR/text_train.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "spectacular-strain",
   "metadata": {},
   "outputs": [],
   "source": [
    "!head $NER_DATA_DIR/labels_train.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "moved-principal",
   "metadata": {},
   "source": [
    "### IOB Tagging\n",
    "We can see that the abstract has been broken into sentences.  Each sentence is then further parsed into words with labels that correspond to the original HTML-style tags in the corpus. \n",
    "\n",
    "The sentences and labels in the NER dataset map to each other with _inside, outside, beginning (IOB)_ tagging. Anything separated by white space is a word, including punctuation.  For the first sentence we have the following mapping:\n",
    "\n",
    "```text\n",
    "Identification of APC2 , a homologue of the adenomatous polyposis coli tumour suppressor .\n",
    "O              O  O    O O O         O  O   B           I         I    I      O          O  \n",
    "```\n",
    "\n",
    "Recall the original corpus tags:\n",
    "```html\n",
    "Identification of APC2, a homologue of the <category=\"Modifier\">adenomatous polyposis coli tumour</category> suppressor .\n",
    "```\n",
    "The beginning word of the tagged text, \"adenomatous\", is now IOB-tagged with a <span style=\"font-family:verdana;font-size:110%;\">B</span> (beginning) tag, the other parts of the disease, \"polyposis coli tumour\" tagged with <span style=\"font-family:verdana;font-size:110%;\">I</span> (inside) tags, and everything else tagged as <span style=\"font-family:verdana;font-size:110%;\">O</span> (outside).\n",
    "\n",
    "# Model configuration\n",
    "\n",
    "Our Named Entity Recognition model is comprised of the pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model followed by a Token Classification layer.\n",
    "\n",
    "The model is defined in a config file which declares multiple important sections. They are:\n",
    "- **model**: All arguments that are related to the Model - language model, token classifier, optimizer and schedulers, datasets and any other related information\n",
    "\n",
    "- **trainer**: Any argument to be passed to PyTorch Lightning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "speaking-grant",
   "metadata": {},
   "outputs": [],
   "source": [
    "WORK_DIR = \"WORK_DIR\"\n",
    "os.makedirs(WORK_DIR, exist_ok=True)\n",
    "MODEL_CONFIG = \"token_classification_config.yaml\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "demanding-ballet",
   "metadata": {},
   "outputs": [],
   "source": [
    "# download the model's configuration file \n",
    "config_dir = WORK_DIR + '/configs/'\n",
    "os.makedirs(config_dir, exist_ok=True)\n",
    "if not os.path.exists(config_dir + MODEL_CONFIG):\n",
    "    print('Downloading config file...')\n",
    "    wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/conf/' + MODEL_CONFIG, config_dir)\n",
    "else:\n",
    "    print ('config file is already exists')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "criminal-outdoors",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this line will print the entire config of the model\n",
    "config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'\n",
    "print(config_path)\n",
    "config = OmegaConf.load(config_path)\n",
    "# Note: these are small batch-sizes - increase as appropriate to available GPU capacity\n",
    "config.model.train_ds.batch_size=8\n",
    "config.model.validation_ds.batch_size=8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "divine-belly",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(OmegaConf.to_yaml(config))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dedicated-effort",
   "metadata": {},
   "source": [
    "# Model Training\n",
    "## Setting up Data within the config\n",
    "\n",
    "Among other things, the config file contains dictionaries called dataset, train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.\n",
    "\n",
    "We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. \n",
    "So, to start model training, we simply need to specify `model.dataset.data_dir`, like we are going to do below.\n",
    "\n",
    "Also notice that some config lines, including `model.dataset.data_dir`, have `???` in place of paths, this means that values for these fields are required to be specified by the user.\n",
    "\n",
    "Let's now add the data directory path to the config."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "informed-purse",
   "metadata": {},
   "outputs": [],
   "source": [
    "# in this tutorial train and dev datasets are located in the same folder, so it is enought to add the path of the data directory to the config\n",
    "config.model.dataset.data_dir = os.path.join(DATA_DIR, 'NER')\n",
    "\n",
    "# if you want to decrease the size of your datasets, uncomment the lines below:\n",
    "# NUM_SAMPLES = 1000\n",
    "# config.model.train_ds.num_samples = NUM_SAMPLES\n",
    "# config.model.validation_ds.num_samples = NUM_SAMPLES"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "changed-mauritius",
   "metadata": {},
   "source": [
    "## Building the PyTorch Lightning Trainer\n",
    "\n",
    "NeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.\n",
    "\n",
    "Let's first instantiate a Trainer object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "computational-battlefield",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Trainer config - \\n\")\n",
    "print(OmegaConf.to_yaml(config.trainer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unique-genre",
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets modify some trainer configs\n",
    "# checks if we have GPU available and uses it\n",
    "cuda = 1 if torch.cuda.is_available() else 0\n",
    "config.trainer.gpus = cuda\n",
    "\n",
    "# for PyTorch Native AMP set precision=16\n",
    "config.trainer.precision = 16 if torch.cuda.is_available() else 32\n",
    "\n",
    "# remove distributed training flags\n",
    "config.trainer.accelerator = 'ddp'\n",
    "\n",
    "trainer = pl.Trainer(**config.trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "overall-literature",
   "metadata": {},
   "source": [
    "## Setting up a NeMo Experiment\n",
    "\n",
    "NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "mathematical-portable",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_dir = exp_manager(trainer, config.get(\"exp_manager\", None))\n",
    "\n",
    "# the exp_dir provides a path to the current experiment for easy access\n",
    "exp_dir = str(exp_dir)\n",
    "exp_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "built-scratch",
   "metadata": {},
   "outputs": [],
   "source": [
    "# complete list of supported BERT-like models\n",
    "print(nemo_nlp.modules.get_pretrained_lm_models_list())\n",
    "\n",
    "# specify BERT-like model, you want to use\n",
    "PRETRAINED_BERT_MODEL = \"biomegatron-bert-345m-cased\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "compact-horse",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the specified above model parameters to the config\n",
    "config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "seeing-geometry",
   "metadata": {},
   "source": [
    "Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation.\n",
    "Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "indoor-france",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_ner = nemo_nlp.models.TokenClassificationModel(cfg=config.model, trainer=trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "genuine-pipeline",
   "metadata": {},
   "source": [
    "## Monitoring training progress\n",
    "Optionally, you can create a Tensorboard visualization to monitor training progress.\n",
    "If you're not using Colab, refer to [https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks](https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks) if you're facing issues with running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "changed-expense",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from google import colab\n",
    "    COLAB_ENV = True\n",
    "except (ImportError, ModuleNotFoundError):\n",
    "    COLAB_ENV = False\n",
    "\n",
    "# Load the TensorBoard notebook extension\n",
    "if COLAB_ENV:\n",
    "    %load_ext tensorboard\n",
    "    %tensorboard --logdir {exp_dir}\n",
    "else:\n",
    "    print(\"To use tensorboard, please use this notebook in a Google Colab environment.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "applied-quality",
   "metadata": {},
   "outputs": [],
   "source": [
    "# start model training\n",
    "trainer.fit(model_ner)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cooperative-michael",
   "metadata": {},
   "source": [
    "# Inference\n",
    "\n",
    "To see how the model performs, we can run generate prediction similar to the way we did it earlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "classical-scientist",
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's first create a subset of our dev data\n",
    "! head -n 100 $NER_DATA_DIR/text_dev.txt > $NER_DATA_DIR/sample_text_dev.txt\n",
    "! head -n 100 $NER_DATA_DIR/labels_dev.txt > $NER_DATA_DIR/sample_labels_dev.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adult-ranking",
   "metadata": {},
   "source": [
    "Now, let's generate predictions for the provided text file.\n",
    "If labels file is also specified, the model will evaluate the predictions and plot confusion matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "twenty-abortion",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_ner.evaluate_from_file(\n",
    "    text_file=os.path.join(NER_DATA_DIR, 'sample_text_dev.txt'),\n",
    "    labels_file=os.path.join(NER_DATA_DIR, 'sample_labels_dev.txt'),\n",
    "    output_dir=exp_dir,\n",
    "    add_confusion_matrix=False,\n",
    "    normalize_confusion_matrix=True,\n",
    "    batch_size=1\n",
    ")\n",
    "# Please check matplotlib version if encountering any error plotting confusion matrix:\n",
    "# https://stackoverflow.com/questions/63212347/importerror-cannot-import-name-png-from-matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "connected-typing",
   "metadata": {},
   "source": [
    "## Training Script\n",
    "\n",
    "If you have NeMo installed locally, you can also train the model with `nlp/token_classification/token_classification_train.py.`\n",
    "\n",
    "To run training script, use:\n",
    "\n",
    "`python token_classification_train.py model.dataset.data_dir=PATH_TO_DATA_DIR PRETRAINED_BERT_MODEL=biomegatron-bert-345m-cased`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "legitimate-electric",
   "metadata": {},
   "source": [
    "The training could take several minutes and the result should look something like\n",
    "```\n",
    "[NeMo I 2020-05-22 17:13:48 token_classification_callback:82] Accuracy: 0.9882348032875798\n",
    "[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 weighted: 98.82\n",
    "[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 macro: 93.74\n",
    "[NeMo I 2020-05-22 17:13:48 token_classification_callback:86] F1 micro: 98.82\n",
    "[NeMo I 2020-05-22 17:13:49 token_classification_callback:89] precision    recall  f1-score   support\n",
    "    \n",
    "    O (label id: 0)     0.9938    0.9957    0.9947     22092\n",
    "    B (label id: 1)     0.8843    0.9034    0.8938       787\n",
    "    I (label id: 2)     0.9505    0.8982    0.9236      1090\n",
    "    \n",
    "           accuracy                         0.9882     23969\n",
    "          macro avg     0.9429    0.9324    0.9374     23969\n",
    "       weighted avg     0.9882    0.9882    0.9882     23969\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unique-fiber",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
